Improving the I/O Throughput for Data-Intensive Scientific Applications with Efficient Compression Mechanisms

Today’s science is generating significantly larger volume of data than before. Data compression can potentially improve application performance. However, in many scientific applications and especially in large scale parallel scientific applications, each process often just accesses parts of the data. This can result in some data that are decompressed by a process but not used. General compression libraries (e.g. LZO [1], bzip2 [2] and zlib [3]) do not consider chunk size in the context of parallel and distributed file systems. Indexing is a widely used technique for online scientific encoding and query e.g. [4–6], even though it would not yield a high compression ratio for large chunks. Some mechanisms [7–9] were proposed by providing user level libraries, indicating the application developers need to modify the application and/or the high-level library. We propose two techniques to leverage compression to improve the performance of large scale parallel applications. First, we enable decompression from the middle of the chunk and stop decompression after we extract the data that we need, which eliminates the overhead of decompressing the data that is not needed by the process. Second, we build compression into the parallel file system which allows caching and prefeteching to be seamlessly integrated and allows applications to transparently leverage compression. Caching decompressed data allows the data to be accessed in later points.